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ABSTRACT 

Designed to present a rationale for evaluating the 
implementation of computer assisted instruction (CAI) in higher 
education, this paper describes a multi-paradigmatic approach to the 
evaluation of CAI pilot implementation efforts and its application in 
various math and English courses at the New York Institute of 
Technology (NYIT) . Students in two math courses — Developmental Math 
and College Algebra — and four different English 

courses— Communication Skills, Composition, Business Writing, and 
Technical Writing— spent significant portions of class time working 
m a microcomputer laboratory utilizing Apple/Franklin and Commodore 
64 computers. Evaluations of instructional effectiveness focused on a 
process analysis, but a quantitative evaluation was also conducted. 
Included are discussions of: (1) CAI and student role changes; (2) 
CAI and faculty role changes; (3) faculty professional development; 
(4) points unique to the math experience; (5) points unique to the' 
English experience; (6) student attitudes toward computers and math 
CAI; (7) students' perceptions of word processing in the learning of 
writing; and (8) unanticipated negative consequences. Findings 
indicate that: CAI students demonstrated a good conceptual grasp of 
subject matter; their difficulties were identified quickly; they 
often worked in pairs, which provided mutual support; and faculty 
were able to interact individually with students, observe the 
learning process closely, and intervene directly and immediately. 
( JB) ^ 
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INTRODUCTION 



The purpose oi -tjhis paper is to present a rationale for' - 
'avaluatin^ thje implementation of. CAI in higher education. We 
refer to this rationale as multi -paradi gmatic since it seeks to 
inte-grate multiple paradigms or approaches to evaluation 
research. Briefly, we are arguing for the complementary uses 
of experimental and case study designs, the merging of outcome 
and process analyses, and obtaining qualitative as well as 
quantitative data. Furthermore, change is assessed, at various 
levels — including student, faculty, and institutional. 

We view computer-based interventions as complex and systemic 
in nature. They affect basic teaching-learning roles, curriculum 
contents, and the very structure of educational delivery." 
Therefore, we feel that it xwi«aintial to use multiple research 
desi^gns, data sources, and assessment criteria. It is especially 
important that the research questions seek to obtain information 
on so-called "unanticipated consequences" (both positive and 
negative) of CAI interventions. 

We will illustrate the application of our multi-paradigmatic 
approach to the evaluation of the CAI pilot implementai on 
efforts in various math and english courses at N.Y.I.T., in 
connection with the Title III grant from the USDDE. Our 
objective is to convey the evolutionary nature of our approach. 
That is, it is not a fixed model or finished product. Our ideas 
about how to approach the issue of CAI evaluation are constantly 
changing with additioanl e::perience. 
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During the 1983-84 academic year CAI was introduced into two 
math courses-Oevelopmental Math and College Algebra (3004 and 
3011) and four different English courses-Communication Skills, * 
Composition,. Business Writing and Technical Writing (1005,1010 
1042, and 1043). ^ 

Significant portion^ of class'^time were held in the\^ 
microcomputer laboratory. Math students used Drill and Practice 
programs and LOGO based problem solving on Apple/Franklin 
computers. English students used word processing -software on 
Commodore 64 computers. "Free Acess" hours to the lab were 
allowed for student assignments. 

Given the- wide variations- of faculty implementation of CAI 
and diverse student backgrounds, we decided to focus our 
evaluation on a process analysis. However, in keeping with the 
multi-paradigmatic rationale, we also conducted a quantitative 
evaluation to complement the process evaluation. We begin our 
discussion" with the process analysis. 
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PROCESS ANALYSIS 
Research Procedures 

Four orimary sources oi data provided the basis for our 
process analysis: <1) faculty interviews, (21 faculty end-term 
experiential questionnaires, (3) student end-term experiential 
questionnaires, and (4) laboratory observations of CAI in action 
These data give a sense of the rich contextual nature of 
computer-based interventions and, we believe, suggest guidelines 
for implementing computer-based projects in other settings an4 

» 

in dif-ferent disciplines. 

Findings 

We have selected some of the more salient findings 
pertaining to both Math and English^ 

. CAI and Student Role Changes, Faculty dbserved the. 
fol lowing changes in student role as a consequence of the CAI 
experience: — 

* CAI students demonstrated a better conceptual grasp of 
subject matter. 

t CAI induced a more active and participatory role in 
learning. 

* CAI students asked more questions. 

* Students reported a sense of mastery due to their ability 
to control the computer. 

« Greater perseverance at math problem-solving and at 
revising and editing English writing assignments was 
reflected in many students' working after lab sessions 
were over. 
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* Student di-fficulties were often identified sooner than 
usual and addressed more quickly. 

* CAI encouraged more independent work. 

* At the same time, CAI studeats often worked in pairs, 
providing peei — based mutual support. 

* Peer-teaching led to greater- social cohesiveness and 
created a positive social climate among students. 

Some of the above changes were due to the uniqueness of the 
computer experience itself, while others (e.g., increased peer 
support) were more probably ' consequences of the lab setting. 

CAI and Faculty Role Changes . The CAI experience led tc 
changes in faculty role as well: 

* The lab experience allowed faculty to observe the learning 
process more closei-y acS'to' intervene more directly and 
immediately. 

* A greater number of individual interactions with students 
resulted. 

* New ways of explaining important concepts emerged as" as 
consequence of the computer's requirement that problems 
be broken down into logical segments. 

* In lab setting, the professor tended to become more a 
facilitator of learning than an authoritative lecturer. 

It is clear that these faculty changes are potentially 
far-reaching and are likely to translate into new patterns of 
faculty-student interactions, besides stimulating more active 
student learning. 
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Faculty Pro-fess ional Development , The impact o-f the CAI 
experience on faculty development was multi -faceted and dramatic. 
Some consequences of Title III participation were: 

* Learning a new teaching skill, i.e., CAI. 

* Gaining new insights into the teaching of math problem 
solving and the writing process. ' 

» Feeling a sense of rejuvenation about teaching mathematics 
and writing, especially in the difficult devel opmenta/ 
courses. 

* Participating in the design of original microcomputer 
software and accompanying instructional materials. 

* Participating in research on CAI. 

* Acquiring some programming skills. 

* Achi^ing visibility an4.3«antng recognition on campus for 
computet — related activities, 

* Gollaborating with collegues -from other disciplines. 

* Presenting conference papers on their CAI experiences. 

Since faculty are the primary mediators of students learning^ 
their new professional development will, in the long run, 
contribute to improved student learning. Furthermore, we view 
such faculty development as crucial in catalysing institutional 
change. Such change will, no doubt, include larger-scale 
adoption of computer-based educational delivery systems. In 
turn, these new delivery systems should continue to facilitate 
student learning. 
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P.pint5 Unique to the Math Eicperience. 

* Commercial software was excellent for first-semester 
developmenl^ math students. Its ease of use and structured 
ftirf^.ats provided an excellent introduction for 
inexperienced students. As students became experienced 
with CAI and more sophisticated mathematically, commercial 
programs were of less interest. The drill-and-practice 
nature of such programs has timer-limited appeal. 

* LOGO, which offers possibilitias for free exploration, 
filled the gap of falling studlSt^/^^^ScIf 
dimension in CAI. 

Points Unique to the "Eng-Hsh Experience. 

* The use of word processing added a new kind of "writing, 
workshop" intta the course. 

* Student writing became more public on the computer monitor 
and more accessible to change during the writing process; 
faculty moved from station to station providing immediate 
feedback. 

* Students tended to produce longer and more ambitious 
papers and carried their papers through more revisions 
than in a conventional course. 

* The longer essays and greater number of revisions placed 
an extra burden on the faculty. " 

Q BEST COPY AVAILABLE 



student Attitudes Toward Computers and Math CAI. 

Students' attitudes toward computer use were, overall, 
very positive. Several general observations emerged from their 
responses to the questionnaire. 

* The overwhelming majority of students saw the computer 
as helplful, useful, and necessary. 

* The computer was viewed as providing a structure for 
organinzing learning — requiring studoets to proceed - 
sequentially and carefully through the problem-solving 
process. . 

* CAI permitted more opportunities for practice on math 
problems and exercises* 

* CAI was seen as providing immediate feedback — reinforcing 
correct responses, pointing out errors, and guiding 
correct problem-solving. 

* Computer exercises assisted in studying and in preparing 
■for exams. 

* The computer appeared to stimulate greater perseverance, 
yielding longer task-pei^f ormance periods. 

« The computer appeared to engender greater interacti^vity , 
responsibility, and accountability in the learning 
process. 

* Some students felt "burdened" at the outset of the CAI 
experience by the extra demands placed on them. 
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Students Perceptions on Word Processing in the L earning of 
Writing , 

The following patterns of observation emerged. 

« Word processing aided mostly in revising and editing. 

* It was viewed as making writing more fun. 

* It helpeci students pull ideas together. 
» It was seen as making writing easier. 

* It saved time. 

* Word processing made grammar and spelling errors more 
^ . visible. 

« It allowed concurrent comparisons of multiple versions. 
Unanticipated Negative Consequences 4^ 



* Hardware problems continually plagued the English facuTQ 
especially, costing class time and causing much .3 
frustration oyer the unnecessary loss of student worl<; 

* Weaker students required much guid^ce in learning to* 
work with the hardware, the menu-driven math so-ftware, 
and word processor. ^ 

* Free exploration of LOGO applications proved difficult far 
weaker students. 

*. The development*! students (1005) took longer to learn word 
processing than those whose language skills were already 
at college level. A significant number (33'/.) required 
over 4 weeks. 
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* Mo^t of the advanced students (those in the technical and 
business writing course) required fewer than 4 weeksi 

* Likewise for math, developmental students (those in 3004) 
took longer than more advanced students (those in 3011). 

* ^^pproKimately SO'/L of- developmental students required 4 
weeks for computer comfort, while iz'L required more than 

* 4 weeks. 

* The majority of advanced students required fewer than 4 
weeks for computer comfort. - 
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QUAN;'Il'AfIVE ANALYali 



The purpose of the quantiiacive pilot studies was to 
de-teroine -she sensiiivioy of our assessment ins , rumen bs, 
the appropria-teness of the dimensicnw (variables) 
targeted for assessment ^ and the utility of our quasi- 
experimental research design. 

Research Proc edures 

♦ 

Quantitative data were obtained within the context of a 
quasi—experimental design* Where feasible, we employed addi- 
tional sections of the same course not exposed to CAI to serve as 
cbmparrison groups. No special designations were noted in the 
college course catalogue to indicate « CAI section, thus mini- 
mizing self-selection to the extent possible. 

* 

The following is a listing of the measures obtained on 
students in the experimental and comparison classes. Most of the 
infbirmation was gathered through administration of an assessment 
package during the first (pre) and last (post) weeks of the 
semester. Final grades and retention data were obtained 
archivally (i.e., from official college records). 
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TABLE I 
LIST OF STUDENT VARIABLES^ 



Mathematics 

Atjtitude Towatds Computer Scale (Pre and Post) 

Computer Literacy Test (Pre and Post) 

Attitude Towards Math Scale (Pre and Post) 

Diagnostic Exam (Pre and Post) 

Final Grades (Post) 

Retention (Post) 

Demographic Infoincnation (Pre) 



English 

Attitude Towards Computer Scale (Pre and Post) 

Computer Literacy Test (Pre and Post) 

Attitude Towards Writing Scale (Pre and Post) 

Pinal Grades (Post) 

Retention (Post) 

Demographic Information (Pre) 
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Findings 

Math 

A significant pre-post difference emerged on the Math 
Diagnostic Test for Math 300^ » indicating a positive gain 
in knowledge* Only :his one experimental class was 
administered che pre-posV diagnostic exam during iJpring^ 
198^. In the current semester, Fall, 198^^ all CAI and 
conventional math classes were administered a revised 
diagnostic ex^. i?his will make our current outcome 

research more meaningful. 

14 



No significant pre-post changes emerged between the 
CAI- and non CAI classes on the Attitude Toward Computer- and Math 
Scalejand the Computer Literacy Test. 

We compared grades and retention for Spring, 198^, 
math CAI and conventional classes. The mean grade for CAI 
classes, was 2,54 versus Z.iZ for comparison classes, The 
percentage of course withdrawals for CAI classes was 27?5» 
compared to 36.8Z% for non-CAI classes. We have refrained 
from, a sratistical analysis » because the conventional and 
CAI classes were not equivalent. However, the trend does 
appear to favor the CAI intervention, 

English 

No significant pre-posr changes emerged between the 
CAI and non CAI classes in the Attitude Towards Wribing 
Scale, Attitude tov?ard Computer Scale and the Computer 
Literacy Scale • 

Concerning grades and retention the data was not 
conclusive* The mean grade for CAI classes was 2.^7 versus 
2»37 for comparison classes. Phe percentage of course 
withdrawals for CAI classes was 18 .S^, compared to 15*99 
for non-CAI classes. 

Wo cannoc draw any conclusions ftom these daoa because 
the groups were not truly comparable* Furthermore among-^-^the 
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CAI classes widely divergent soudent populations were 
represented and ins ructor interventions were not 
standardized. 

We did not include a s andard diagnostic measure last 
year. However, we are experimenting with a Standard Writing 
Diagnos-cic Measure during this curxent-Fall,. 198^ ^semester, 
ffisdcing outcome research more meaningful • 

Discussion 

Tlia. issue of grades as an index of outcome in measuring 
the effects of CAI needs to he addressed ac this points 
There is no clear basis to expect overall improvement in grades 
cue tc Che CAI intervention because the intervention only 
eficompassed a fraction of class time and workr while grades^ are 
iased on 100% of the worlc^ 

Furthermore, assignment of final couirse grades do not adhere 
to a standard criteria across classes, as different professors 
vary in- their grading procedures,. 

In addition, since CAI and non-CAI classes cannot be matched 
in a quasi-experimental design > there is no control group to 
provide a true basis of comparison. EV^en more so^ the conditions 
cannot even be created through random assignment.. Thus, 
differences between groups exist on such variables as native 
ability, motivation^ time available to study, personal problems^ 
and interest in CAI. The above factors are more likely to 
contribute to attrition more so than being in a CAI or non-CAl 
class • 16 



The nature of our CAI intervention is such that it does not 
"teach to test," Rather, a wore general problem solving tool (in 
the case of math) or writing tool (for English) constitutes the 
core of the intervention. It follows, therefore, that grades ar^ 
a very narrow indicator of CAI success. This rationale also 
•applies to- the diagnostic test (reported for math). We plan to 

test the impact o£ CAI on creative problem solving,, on learning 
to iearn, and on conceptual abilities.^ 

Concerning the attitude measures two major factors can 
explain the general lack of between group difference. First, the 
intervention, was not geared to inducing such attitudinal changes ► 
Second, our instruments may not have been sensitive enough to 
detect subtle changes in attitude which have occurred ^as we 
discussed earlier' in this report.. 

Also, it should be pointed out that great variations existed 
in the implementation of CAT among the professors^ This was 
purposely left unstandardized so that each faculty member could , 
adopt his/her own style for experimenting with CAI. This fact 
also made it difficult to rigorously evaluate outcome at this 
time.. 

Finally, concerning retention, we feel that the prescence of 
CAI will, over time, attract more qualified students to NYIT, and 
thereby reduce attrition. , 
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in conclusion, we feel that CAI shows great promise for 
teaching students both academic and real-world work slcills. As 
career oriented institution it is to our advantage to continue 
the incorporation of CAI in various, disciplines. 

The microcomputer is emerging as a powerful aid for ^ 
presenting and organizing factual material across many 
disciplines. AS a more advanced problem-solving tool, the 
features of the computer are indeed unique and require our 
further exploration. 
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